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Abstract 

The  CoAX  demonstration  was  an  international  effort,  sponsored  by  the  Defense  Research  Projects 
Agency  (DARPA),  Defense  Science  Technology  Office  (DSTO),  Defense  Scientific  Technical 
Laboratory  (DSTL),  and  The  Technical  Cooperation  Program  (TTCP).  CoAX  was  a  collaboration 
between  DoD  labs,  industry,  and  academia  to  showcase  the  power  of  software  agents  to  rapidly 
construct  and  maintain  a  coalition  Command  and  Control  Structure.  The  CoAX  demonstration 
was  conducted  through  numerous  Technology  Integration  Experiments  (TIEs).  The  purpose  of  the 
TIEs  was  to  pair  up  participants  in  the  demonstration  such  that  synergistic  effects  of  software 
agent  technology  being  developed  by  each  participant  could  be  demonstrated.  The  Naval  Research 
Laboratory  and  the  University  of  Maryland  participated  in  a  TIE  which  demonstrated  the 
integration  of  the  NRL  Global  Command  and  Control  System  Maritime  ( GCCS-M)  surrogate  with 
the  Interactive  Maryland  Platform  for  Agents  Collaborating  Together  (IMPACT)  system.  This 
paper  will  describe  the  CoAX  demonstration,  the  TIE  that  was  conducted  between  NRL  and  UMD, 
and  future  interactions  that  are  designed  to  facilitate  the  integration  of  multimedia  information  into 
GCCS-M  via  IMPACT. 
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1  Introduction  to  Coalition  Operations 

Military  coalitions  are  complex  organizations  that,  in  many  cases,  must  be  rapidly  created  and 
effectively  managed  as  the  battlefield  dynamics  change.  Issues  such  as  information  security, 
interoperability  between  data  and  systems  across  geopolitical  boundaries,  lack  of  information,  and 
labor-intensive  approaches  to  data  collection  are  a  few  examples  of  barriers  that  must  be  overcome 
in  order  for  the  coalition  to  be  successful  in  meeting  its  objectives.  Systems  generally  exist  in 
stovepipe  form,  particularly  across  coalition  boundaries,  and  it  is  very  difficult  for  them  to 
exchange  meaningful  data.  Not  only  must  a  coalition  be  concerned  with  data  interoperability 
among  its  own  forces  and  be  able  to  react  to  an  adversary  on  the  physical  battlefield,  but  it  must 
also  have  the  technology  in  place  to  react  appropriately  in  an  era  when  attacks  can  be  carried  out  in 
cyberspace. 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 
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The  U.S.  Navy’s  vision  of  FORCENET  is  to  integrate  networks  of  sensors,  weapons,  systems,  and 
platforms  together  in  order  to  multiply  force  power.  If  this  is  to  become  a  reality,  such  systems 
must  be  easily  integrated  to  share  information  in  ways  not  imagined,  such  that  barriers  associated 
with  the  current  nature  of  being  stovepipe  to  data  and  other  systems  are  overcome.  Software  agents 
are  being  increasingly  examined  as  a  potential  technology  to  overcome  some  of  the  barriers  in 
making  EORCENET  a  reality,  particularly  across  a  coalition  command  and  control  structure. 

Agent  aided  information  retrieval  and  decision  support  has  attracted  the  attention  of  the  agent 
research  community  for  several  years.  The  concept  of  large  ensembles  of  semi-autonomous, 
intelligent  agents  working  together  is  emerging  as  an  important  model  for  building  the  next 
generation  of  sophisticated  software  applications.  This  model  is  especially  appropriate  for 
effectively  exploiting  the  increasing  availability  of  diverse,  heterogeneous,  and  distributed  on-line 
information  sources,  and  as  a  framework  for  building  large,  complex,  and  robust  distributed 
information  processing  systems.  The  development  of  enabling  infrastructure  for  mobile  computing 
and  interoperability  among  programs  residing  at  distant  sites,  and  new  generations  of  distributed 
operating  systems,  will  continue  to  make  the  construction  of  systems  based  on  this  model  much 
easier.  Software  agents  represent  a  new  paradigm  in  distributed  computing.  The  notion  of 
autonomous  software  entities  able  to  work  autonomously,  or  in  cooperation  with  each  other,  to 
perform  tasks  in  satisfying  its  objectives  represents  a  powerful  concept.  Thus,  software  agents 
have  been  deployed  in  many  commercial,  academic,  and  military  domains. 

In  the  following  sections,  we  will  describe  the  CoAX  30-month  experiment,  explain  the  TIE 
interaction  between  NRL  and  UMD  to  support  this  demonstration,  and  provide  details  on  future 
collaboration,  particularly  building  upon  the  TIE  in  order  to  leverage  the  multimedia  capabilities  of 
IMPACT  to  potentially  enhance  the  GCCS-M  Common  Operational  Picture  (COP). 


Figure  1:  The  fictional  countries  used  in  the  CoAX  demonstration 


2  Multi- Agents  Systems  Support  for  Coalition  Agents  Experimentation  (CoAX) 

The  Defense  Advanced  Research  Projects  Agency  (DARPA)  has  sponsored  the  Control  of  Agent 
Based  Systems  (CoABS)  program  [9],  one  of  their  largest  programs  investigating  the  use  of 
software  agents  in  support  of  military  command  and  control.  The  Naval  Research  Laboratory 
(NRL)  has  participated  in  the  Coalition  Agents  Experiment  (CoAX)  [10] 
(a  culmination  of  the  CoABS  program),  an  international  collaborative  effort  between  DARPA, 
Defense  Science  Technology  Organization  (DSTO),  Defense  Science  Technology  Laboratory 
(DSTL),  and  The  Technical  Cooperation  Program  (TTCP).  In  addition,  DoD  laboratories, 
universities,  and  industry  provided  software  agent  research  and  technology  in  order  to  demonstrate 
that  multi-agent  systems  are  an  effective  mechanism  in  fostering  coalition  interoperability.  The 
scenario  chosen  for  CoAX  was  based  on  operational  concepts  developed  by  TTCP  describing  the 
fictional  country  of  Binni  shown  in  Figure  1.  In  the  scenario,  two  fictional  countries,  Agadez  and 
Gao  are  disputing  over  Binni  due  to  the  geopolitical  and  economic  development  of  this  fictional 
country.  From  an  operational  point  of  view,  a  coalition  is  formed  to  bring  peace  to  the  region,  and 
this  coalition  is  brought  together  through  software  agents. 

Each  of  the  software  agents  (approximately  70)  representing  various  functional  capabilities  of  the 
coalition  were  incorporated  within  military  domains,  for  example,  the  coalition  forces  maritime 
component  command  in  Figure  2.  The  interoperability  of  the  agents  was  enhanced  through  the  use 
of  ontologies  which  provided  semantic  meaning  to  the  information  that  was  exchanged  by  the 
agents.  Associated  with  each  domain  were  policies  that  provided  bounds  on  agent  behavior.  The 
enabling  technology  developed  under  the  CoABS  program  and  utilized  for  the  CoAX 
demonstration,  the  CoABS  agent  grid  [9],  provided  the  infrastructure  to  allow  the  agents  to 
interface  and  exchange  information  with  each  other.  The  CoABS  agent  grid  is  currently  being 
utilized  in  joint  experimentation  programs  such  as  the  Navy  Expeditionary  Sensor  Grid  Enabling 
Experiments  (the  ESG  demonstrates  the  utility  of  flexibly  and  dynamically  integrating  naval 
sensors  and  C2  systems  via  the  agent  grid),  the  Air  Force  Joint  Battle  InfoSphere,  and  several 
Army  programs. 


Figure  2:  The  military  domains  associated  with  the  CoAX  demonstration 

The  CoAX  demonstration  was  conducted  through  Technology  Integration  Experiments,  or  TIEs. 
The  purpose  of  each  TIE  was  to  pair  up  certain  groups  based  on  the  synergistic  nature  of  the 
technology  solutions  provided  by  each  of  the  participants  in  support  of  the  Binni  Scenario. 


3  NRL  TIE  with  UMD 


The  purpose  of  the  TIE  collaboration  between  the  University  of  Maryland  (via  their  IMPACT 
system)  and  the  Naval  Research  Laboratory  (through  their  Global  Command  and  Control  System 
Maritime  -  GCCS-M  -  surrogate)  was  to  demonstrate  the  capability  of  tracking  and  predicting  the 
location  of  enemy  submarines  via  agents.  Before  discussing  the  details  of  the  TIEs,  it  is  important 
to  set  the  stage  by  describing  what  transpired  in  the  Binni  scenario. 

In  the  scenario,  both  Agadez  and  Gao  are  in  dispute  over  Binni,  with  Agadez  becoming 
increasingly  desperate  over  the  territory.  Because  of  this  desperation,  they  launch  a  missile  strike 
against  an  Australian  ship  in  the  Red  Sea.  This  strike  injures  many  on  board  and  damages  critical 
capabilities  on  the  ship,  including  the  Magnetic  Anomaly  Detection  systems  (MAD  detectors).  As 
the  coalition  becomes  aware  of  the  strike  and  subsequent  damage,  mobile  medical  monitoring 
agents  are  dispatched  to  the  ship  to  collect  injury  reports  contained  in  medical  databases.  Erom 
these  reports,  agents  are  able  to  work  cooperatively  to  schedule  the  evacuation  of  those  critically 
injured.  A  neutral  country  on  the  eastern  coast  of  the  Red  Sea,  Arabello,  agrees  to  offer  its  ASW 
(Anti-Submarine  Warfare)  capabilities  to  track  down  and  neutralize  the  Agadez  submarines.  These 
ASW  capabilities  are  rapidly  integrated  via  agents  with  the  coalition  systems  already  in  place,  and 
expeditiously  begin  to  provide  contact  reports  on  the  possible  locations  of  the  Agadez  submarines. 
With  regard  to  Eigure  3,  the  Information-Trust-Evaluator  (ITE)  agents  from  the  University  of 
Texas  assign  a  confidence  value  to  each  ASW  report.  Once  these  confidence  values  have  been 
assigned,  NRL  database  agents  communicate  with  the  ITE  agents  to  pull  and  store  the  reports  in  the 
simulated  Track  Database  Manager  (TDBM)  server  for  eventual  display  in  the  GCCS-M 
“surrogate”  map  view.  The  user  is  then  able  to  interact  with  the  map  view  to  request  “predictions” 
on  the  future  state  of  the  tracks  from  the  UMD -Predict  agent. 
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Figure  3:  Several  TIE  components  for  Co  AX 


3.1  NRL  Technology  offered  for  Co  AX 


The  NRL  participated  under  the  military  domain  of  Coalition  Forces  Maritime  Component 
Command  (CFMCC  -  a  major  component  of  CoAX),  and  interfaced  a  GCCS-M  surrogate  to  the 
agent  grid.  This  demonstrated  the  manner  in  which  software  agents  would  integrate  information 
into  the  Navy  system.  Furthermore,  the  purpose  of  the  demonstration  was  to  show  how 
commanders  could  interact  with  such  an  agent-enabled  system  to  make  decisions.  The  surrogate 
GCCS-M  system  was  a  COTS  product  called  the  extensible  Information  System  (XIS)  [12].  The 
XIS  was  chosen  for  its  close  functionality  to  GCCS-M,  and  is  planned  as  a  future  segment  within 
the  Defense  Information  Infrastructure  (DII)  Common  Operating  Environment  (COE). 

Using  the  GCCS-M  surrogate  and  map  view,  a  user,  representing  a  CEMCC  staff  member,  was 
able  to  request  a  prediction  on  the  Agadez  submarine  tracks  from  the  IMPACT  agents.  Using  the 
map  view,  a  user  was  able  to  draw  a  rectangular  region  comprised  of  an  array  of  cells  (this  array  of 
cells  can  be  seen  in  the  left  image  of  Eigure  4).  This  information  was  communicated  by  the  NRL 
display  agent  to  IMPACT  agents  which  returned  a  probabilistic  value  associated  with  each  cell. 
These  values  signify  the  potential  future  locations  of  the  submarines  at  a  user  defined  time-instant 
or  time-window.  This  can  be  seen  in  the  right  image  of  Eigure  4.  In  this  image,  one  can  see  a  two- 
dimensional  probability  distribution  in  the  Red  Sea  which  is  returned  by  IMPACT.  The  higher 
(resp.  lower)  probabilities  are  shown  in  red  (resp.  cyan). 


Figure  4:  Prediction  cells  that  are  sent  to  the  UMD-Predict  agent  (left)  and 
the  results  returned  by  the  UMD-Predict  agent  (right) 


3.2  Description  of  the  UMD- Predict  Agent 


The  aforementioned  probability  distribution  was  generated  by  the  UMD-Predict  agent.  This 
agent  was  written  in  IMPACT,  the  Interactive  Maryland  Platform  for  Agents  Collaborating 
Together  [1,  7,  8].  IMPACT’S  powerful  libraries  made  it  easy  to  interpret  messages  from  the 
CoABS  agent  grid  and  to  efficiently  manipulate  the  times,  positions,  velocities,  confidence  values, 
probabilities,  etc.  contained  in  these  messages  in  order  to  derive  reasonable  predictions. 

There  is  a  large  variety  of  models  that  can  be  used  to  generate  predictions.  Some  of  these  models 
infer  future  states  based  on  past  observations,  for  example,  by  fitting  a  curve  through  the  positions 
of  an  Agadez  submarine  that  were  reported  by  an  Information-Trust-Evaluator  agent.  Other 
models  may  create  forecasts  based  on  the  present  situation  without  considering  the  past.  For 
example,  based  cn  the  asset  locations  and  priorities  specified  by  a  QinetiQ -Intel  agent  (another 
agent  on  the  grid  providing  intelligence  reports),  one  of  these  models  may  predict  that  an  Agadez 
submarine  will  head  toward  the  closest  high-priority  asset,  regardless  of  how  the  sub  arrived  at  its 
present  location.  Even  if  we  restrict  our  attention  to  models  based  on  curve  fitting  for  the 
parametric  equations  X-f{t)  and  Y=g(t),  we  can  still  derive  a  large  number  of  mutually  inconsistent 
estimates.  For  example,  predictions  based  on  choosing  the  best  fitting  line  will  not,  in  general, 
coincide  with  predictions  based  on  choosing  the  best  fitting  parabola.  The  UMD-Predict  agent 
considered  models  based  on  linear,  quadratic,  cubic,  and  periodic  (sine  wave)  regression.  Also,  to 
help  prevent  overfitting  of  the  data,  the  agent  supports  models  that  consider  only  the  n  most  recent 
observations  whose  confidence  value  exceeds  a  threshold  c.  For  example,  model  mi  may 
determine  f(0  and  g(t)  via  linear  regression  over  all  past  observations  while  model  m2  may 
determine  f(t)  via  linear  regression  and  g(t)  via  quadratic  regression  over  the  last  five  reports  that 
came  from  sources  with  a  reliability  of  at  least  90%.  Furthermore,  note  that  the  design  of  UMD- 
Predict  makes  it  easy  to  incorporate  new  models. 

There  are  many  ways  to  combine  a  set  of  models  into  a  single,  unified  model  M.  For  simplicity, 
the  UMD-Predict  agent  only  considered  one  of  these  methods.  Specifically,  based  on  several 
factors  such  as  user-definable  parameters  and  the  sum  of  the  squared  residuals,  each  model  m  was 
assigned  a  weight  p(m)  >  0  such  that  Em  P(^)  =  1-  Then,  for  each  model  m,  each  location  (v,  y), 
and  each  future  instant  t,  define  p(m,  v,  y,  t)  as  p(m)  times  the  probability,  according  to  m,  that  the 
submarine  will  be  at  (v,  y)  during  t.  Thus  Ex  p(^,  y,  0  =  P(^)  for  each  model  m  during  each 
instant  t.  Next,  define  p(M,  v,  y,  t)  as  Em  P(^.  0  for  oach  triple  (v,  y,  t).  It  is  easy  to  verify  that 

Ex  p(^,  f)  =  \  during  each  instant  t.  In  other  words,  the  unified  model  M  can  be 

represented  by  a  set  of  probability  matrices  where  each  matrix  is  associated  with  a  time  interval. 
This  set  of  matrices  can  be  concisely  stored  in  a  probabilistic  temporal  relation  [2].  Note  that  the 
results  returned  by  UMD-Predict  are  obtained  by  modifying  M.  For  example,  if  the  sub  cannot 
reach  (v,  y)  by  time  t  or  if  the  elevation  at  (v,  y)  is  too  high  for  the  sub,  then  p(M,  v,  y,  t)  must  be 
zero.  Furthermore,  it  is  usually  desirable  to  smooth  the  probabilities  in  each  matrix.  Theoretically, 
any  modification  is  acceptable  as  long  as  Ex  Y.y  p(M,  y,  0  remains  unchanged  for  each  t. 

Ordinarily,  UMD-Predict’ s  clients  are  only  interested  in  a  small  subset  of  M.  For  example,  a  query 
may  request  the  portion  of  the  probability  matrix  for  vehicle  v  at  time  t  that  lies  within  a  bounding 
box  b.  Also,  the  client  may  only  be  interested  in  locations  that  can  be  covered  by  an  available 
“sub-killer”  at  time  t  (the  position,  maximum  speed,  etc.  for  these  sub-killers  are  provided  by  the 
QinetiQ-Intel  agent).  Additionally,  to  save  bandwidth,  clients  may  only  be  interested  in 
determining  the  coordinates  of  at  most  n  nonoverlapping  windows  of  size  winX  x  winY  such  that 
the  sum  of  the  probabilities  in  each  window  exceeds  some  threshold  p. 


To  summarize,  UMD -Predict  is  an  agent  that  benefits  from  IMPACT’S  ability  to  efficiently  handle 
multimedia  data  such  as  sensor  reports.  The  forecasts  are  obtained  by  generating  a  large  number  of 
prediction  models,  combining  their  estimates  into  one  unified  model,  transforming  this  model  (e.g., 
by  considering  terrain  and  routing  information),  and  then  filtering  the  results  in  order  to  return 
succinct  yet  relevant  answers.  For  Co  AX,  these  answers  were  visualized  by  NRL  agents. 

The  next  sections  will  describe  technology  being  developed  under  a  separate  effort  that  will  be 
leveraged  to  allow  agents  to  discover  and  integrate  multimedia  information  into  systems  such  as 
GCCS-M  and  ITEM  using  the  CoABS  agent  grid. 

4  Intelligent  Agents  in  the  C4I  to  Simulation  Interface 

Having  integrated  a  GCCS-M  surrogate  in  the  Co  AX  demonstration,  NRL  is  currently  conducting 
experiments  [5]  with  the  actual  GCCS-M,  through  its  integration  with  the  Integrated  Theater 
Engagement  Model  (ITEM)  [5]  shown  Eigure  5.  The  purpose  of  this  integration  is  to  develop  and 
utilize  software  agents  to  decompose  planning  information  and  efficiently  monitor  those  plans 
within  ITEM.  This  work  is  being  sponsored  by  the  Defense  Modeling  and  Simulation  Office 
(DMSO).  The  hypothesis  is  that  agents  integrated  with  both  GCCS-M  and  ITEM  will  improve  the 
generation  of  courses  of  action  (COA)  and  analysis  since  these  agents  will  help  the  user  to 
understand  the  important  cause/effect  relationships  within  the  plans  via  simulation.  The  planned 
demonstration  will  involve  integration  of  the  GCCS  Ambassador  [6],  permitting  tracks  resident 
in  the  TDBM  to  be  published  to  the  High  Level  Architecture  (HLA)  Run-Time  Infrastructure 
(RTI)  [3].  The  real  and  simulated  tracks  will  be  presented  to  agents  on  the  g'id  through  the  Critical 
Mission  Data  over  RTI  (CMDR)  system  [5],  which  bridges  the  HLA  RTI  to  the  CoABS  grid. 
Software  agents  are  being  developed  to  communicate  via  the  grid  to  decompose  military  plans  and 
correlate  events  in  those  plans  to  tracks  in  order  to  monitor  critical  events  in  the  simulation. 
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Figure  5:  Agent  mediated  interface  between  GCCS-M  and  ITEM. 


Plan-understanding  agents  registered  on  the  grid  will  be  capable  of  decomposing  external  planning 
information,  for  example.  Operational  Tasking  (OPTASK)  messages  and  Air  Tasking  Orders 
(ATO’s).  This  decomposed  information  will  be  used  to  form  relationships  between  events  both 
spatially  and  temporally,  as  well  as  associating  the  events  in  the  plans  to  their  associated  tracks. 


The  plan- monitoring  agents  will  monitor  both  ITEM  (which  represents  how  entities  should  be 
moving  and  interacting)  and  GCCS  (representing  how  entities  are  actually  moving  and  interacting 
based  on  a  replay  of  a  scenario)  in  order  to  take  note  of  the  differences  that  are  occurring  with 
regard  to  critical  events  and  their  relationships  with  other  critical  events.  In  other  words,  the  agents 
will  monitor  only  those  events  and  relationships  that  are  deemed  of  critical  importance  with  regard 
to  some  measurement  criteria  (e.g.,  rendezvous  points).  By  measuring  deviations  in  those  critical 
events  (and  not  every  deviation  occurring  in  the  simulation,  since  a  local  deviation  does  not 
necessarily  imply  that  a  mission  will  not  succeed),  the  user  will  be  able  to  more  easily  comprehend 
the  simulation.  Furthermore,  the  agents  can  perform  notification  as  the  courses  of  action  are 
performed  when  the  constraints  imposed  by  these  relationships/dependencies  are  violated.  In  a 
complex  scenario,  visual  detection  of  dependencies  will  be  difficult,  and  automation  through 
software  agents  will  be  valuable. 

4.1  Benefits  of  Multimedia  Information  for  GCCS-M 

We  have  described  the  use  of  the  architecture  in  Figure  5  to  provide  plan  information  from 
GCCS/ITEM  to  software  agents  on  the  grid.  However,  t  can  also  be  used  as  a  mechanism  for 
agents  to  gather  multimedia  from  distributed  sources  and  present  it  to  GCCS/ITEM.  The  CMDR 
program  can  be  modified  to  provide  that  information  in  the  “reverse”  direction.  This  is  being 
considered  as  a  means  to  integrate  multimedia  information  from  IMPACT  in  both  C4I  systems  and 
simulation  since  IMPACT  agents  can  already  communicate  through  the  grid. 

The  GCCS-M  system  can  benefit  from  the  integration  of  remote  multimedia  information  such  as 
overlays,  integration  of  video,  imagery,  and  formatted  textual  information.  The  goal  of  GCCS  is  to 
provide  the  operational  users  with  a  consistent  Common  Operational  Picture  (COP),  and  the 
integration  of  such  information  has  the  potential  to  enhance  the  COP  and  provide  increased 
situational  awareness.  Although  some  capabilities  exist  in  the  DII  COE  architecture  (in  which  the 
GCCS-M  is  a  “software  segment”)  to  permit  the  integration  of  a  limited  set  of  multimedia 
information,  there  is  substantially  more  that  exists  on  the  web  as  well  as  in  legacy  systems  that  may 
profoundly  benefit  such  systems. 

From  an  operational  perspective,  the  DII  COE  is  still  predominantly  client-server  based.  The 
military  is  envisioning  a  paradigm  shift  from  the  DII  COE  environment  to  Net-Centric  Enterprise 
Services  (NCES)  [4].  One  of  the  goals  of  NCES  is  to  leverage  commercial  object  technology,  web 
technology,  and  web  services  as  well  as  distributed  computing  technologies  such  as  network 
storage  and  grid  computing.  These  technologies  have  significantly  matured  during  the  past  several 
years  and  will  provide  building  blocks  for  NCES  while  satisfying  the  design  goal  of  a  loose 
coupling  between  a  broad  set  of  applications  with  increased  levels  of  interoperability  and 
information  sharing.  The  vision  of  NCES  can  profoundly  benefit  from  intelligent  agents  that  are 
able  to  discover  and  tap  into  distributed  multimedia  information  sources  across  a  network. 

4.2  Challenges  in  Multimedia  Integration 

There  are  many  challenges  when  dealing  with  the  integration  of  multimedia  information  from  the 
web  or  legacy  systems  within  critical  systems  such  as  GCCS-M.  The  confidence  in  the 
information  must  be  considered,  as  well  as  the  fusion  of  that  information  with  information  from 
other  similar  sources.  For  example,  the  fusion  algorithm  must  take  into  consideration  the  varying 
levels  of  confidence  associated  with  the  information  in  the  fusion  process.  Another  issue  to 
consider  is  that,  based  on  what  the  user  is  interested  in,  how  to  best  present  the  possible  mix  of 
multimedia  information  to  the  user.  For  example,  what  combination  of  information  provides  the 
most  benefit  in  the  context  in  which  the  user  has  requested  it?  Latency  is  another  concern.  For 


time-critical  applications  latency  should  be  minimized,  but  for  rich  multimedia  information  that 
may  be  voluminous,  there  are  formidable  obstacles  that  can  make  it  difficult  to  keep  latency 
sufficiently  low.  These  are  some  of  the  challenges  that  shall  be  addressed  during  the  integration  of 
multimedia  information  into  IMPACT  and  GCCS-M/ITEM. 

4.3  Multimedia  Capabilities  of  IMPACT 

IMPACT  supports  several  mechanisms  for  acquiring  multimedia  information.  For  example, 
IMPACT  agents  can  reason  over  sensor  data  that  is  acoustic  (e.g.,  from  an  ensemble  of 
microphones),  graphic  (e.g.,  from  remotely  triggered  cameras),  gyroscopic  (e.g.,  to  track  movement 
direction),  seismic  (e.g.,  to  supplement  passive  infrared  motion  detectors),  etc.  This  data  can  be 
obtained  from  several  sources  including  mobile  robots,  remote  web  pages,  and  dedicated  hardware 
such  as  Maxim’s  1-Wire  Weather  Station  [11].  The  utility  of  handling  these  kinds  of  data  has  been 
demonstrated,  for  instance,  by  IMPACT  agents  that  warn  interested  users  when  an  engine  will 
probably  fail  (specifically,  when  the  measurements  for  the  vibration,  temperature,  or  rpm  of 
monitored  components  exceed  their  normal  range).  Similarly,  a  dispatch  agent  whose  plans  depend 
on  the  weather  may  ask  a  monitoring  agent  to  send  a  warning  whenever  the  barometric  pressure, 
temperature,  humidity,  or  wind  speed  of  a  region  exceeds  a  set  of  thresholds. 

IMPACT  also  supports  several  mechanisms  for  presenting  multimedia  information.  For  example, 
numerical  sequences  can  be  visualized  by  charts  in  an  Excel  file  that  is  dynamically  generated 
when  an  agent  triggers  the  appropriate  action.  Similarly,  there  are  actions  that  can  dynamically 
create  a  PowerPoint  (PPT)  file  by  merging  slides  from  existing  PPT  presentations,  filtering  out 
slides  that  are  not  of  interest,  and  if  desired,  inserting  objects  into  slide  templates.  Furthermore, 
IMPACT  includes  actions  that  can  play  audio/video  files,  interact  with  a  remote  controlled  robot, 
send  an  email  notification,  update  a  web  page,  pop-up  an  alert  on  a  Palm  Visor,  send  a  message  to 
a  mobile  phone  that  supports  SMS  (Short  Message  Service),  overlay  images  onto  a  map,  etc.  For 
example,  an  IMPACT  agent  can  read  a  GPS  report,  determine  the  relevant  zip  code,  and  then  feed 
that  information  to  a  web  page  in  order  to  display  live  traffic  video  for  an  area  of  interest.  Clearly, 
an  important  asset  of  IMPACT  is  its  ability  to  flexibly  handle  a  large  variety  of  multimedia  data. 

5  Summary 

We  have  described  the  CoAX  demonstration  and  the  technology  offered  by  NRL  and  UMD  that 
showcased  the  manner  in  which  software  agents  could  be  used  to  dynamically  integrate 
information  in  the  form  of  ASW  contact  reports  within  the  GCCS-M  surrogate.  We  have  also 
described  how  a  user  was  able  to  interact  with  the  surrogate  in  order  to  query  the  agents  within 
IMPACT  for  a  prediction  of  the  future  state  of  enemy  submarines.  The  TIE  between  NRL  and 
UMD  was  a  critical  component  in  the  CoAX  experiment  that  demonstrated  how  these  two  systems 
could  be  used  to  end  the  regional  conflict  between  two  disputing  countries. 

The  CoAX  experiment  successfully  showed  how  software  agent  technology  could  be  used  to 
integrate  (sometimes  incompatible)  systems  together,  the  utility  of  policy  and  domain  management 
facilities  to  bound  agent  behavior  and  facilitate  selective  sharing  of  information,  and  how  the  ease 
of  composing  agent  systems  together  lead  to  adaptive  responses  to  changes  and  unexpected  events. 

We  have  also  disclosed  further  development  and  integration  efforts  (leveraging  an  ongoing  DMSO 
effort)  in  order  to  utilize  the  multimedia  capabilities  of  IMPACT  for  enhancing  the  GCCS-M  COP 
as  well  as  the  ITEM  simulation.  Additionally,  we  characterized  several  issues  that  we  plan  to 
address  in  order  to  make  the  integration  plausible  from  an  operational  perspective. 
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